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Guidance  on  DSTO  Analysis  Support  to 
ADF  Campaign  Assessment 


Executive  Summary 

This  report  offers  a  framework  for  the  monitoring  and  evaluation  activities  undertaken 
by  the  Defence  Science  and  Technology  Organisation  (DSTO)  staff  for  the  Australian 
Defence  Force  (ADF)  Campaign  Assessments  (CA).  Responsibility  for  CA  is  owned  by 
the  J5  (Plans)  Branch  in  Headquarters  Joint  Operations  Command  (HQJOC).  In  July 
2010,  DSTO  was  requested  to  provide  support  to  HQJOC  for  the  CA  process  and  this 
request  was  met  by  DSTO  Operations  Support  Centre's  (DOSC)  Operational  Planning 
and  Evaluation  (OPE)  Team,  DSTO's  deployed  Operations  Analyst  teams,  deployed 
Scientific  Liaison  officers  and  CA  fly-away-teams. 

The  purpose  of  a  CA  is  to  support  effective  planning  and  decision  making  by  assessing 
the  accuracy  of  the  intent  and  achievability  of  a  campaign  plan.  ADF  campaign  plans 
specify  a  hierarchy  of  high-level  objectives  and  set  out  the  Effects  (or  progress 
milestones)  that  will  lead  to  achievement  of  those  objectives.  Effects  define  the 
boundaries  for  the  range  of  activities  that  the  Commander  Joint  Operations  (CJOPS) 
plans  to  undertake  and  the  scope  of  influence  he  expects  to  have  in  the  area  of 
operations. 

Given  the  complexity  and  tempo  of  operations,  the  achievement  of  Effects  is  not 
always  clear  and  usually  requires  monitoring  of  multiple  indicators  of  progress.  The 
OPE  Team  assisted  military  planners  to  identify  a  robust  and  comprehensive  set  of 
indicators  for  the  Effects  identified  in  the  campaign  plan,  grouped  as  a  set  of  Measures 
of  Effect  (MoEs).  These  are  monitored  across  regular  assessment  cycles,  and  validated 
against  broader  monitoring  of  the  operating  environment.  In  this  way,  we  moved 
beyond  simply  documenting  what  had  happened  to  date,  and  evaluation  insights 
became  a  more  sound  basis  for  decision  making  on  what  was  likely  and  desirable  for 
the  campaign  plan  going  forward. 

Importantly,  this  framework  differentiated  between  the  monitoring  and  evaluation 
roles.  The  definitions  and  components  of  each  provided  a  checklist  for  quality 
assurance  of  DSTO's  support.  Most  importantly,  use  of  the  framework  provided  a 
common  vocabulary  for  communicating  and  refining  processes  among  the  OPE  team, 
deployed  operations  analysts,  deployed  ADF  Joint  Task  Forces  and  HQJOC. 
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The  two  primary  points  of  emphasis  for  the  guidance  are  that  the  quality  and  viability 
of  CA  is  contingent  upon  the  extent  of  active  client  engagement  throughout  the 
process,  and  the  clarity  and  consistency  with  which  DSTO's  support  to  that  process  is 
delivered.  Adherence  to  the  program  evaluation  framework  as  described  in  this 
guidance  will  facilitate  both  of  these. 
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Glossary 


ADF 

Australian  Defence  Force 

CA 

Campaign  Assessment 

CAWG 

Campaign  Assessment  Working  Group 

CJOPS 

Commander  Joint  Operations 

DIME 

Diplomatic,  Information,  Military,  Economic 

DOSC 

DSTO  Operations  Support  Centre 

DSTO 

Defence  Science  and  Technology  Organisation 

HQ 

Headquarters 

HQJOC 

Headquarters  Joint  Operations  Command 

HQJOCJ5 

Headquarters  Joint  Operations  Command  Plans  Branch 

LARF 

Logic  Assumptions  and  Risks  Framework 

LoO 

Line  of  Operation 

MoE 

Measure  of  Effect 

MoP 

Measure  of  Performance 

OA 

Operational  Analyst 

OGA 

Other  Government  Agency 

OPE 

Operational  Planning  and  Evaluation 

OPSTSR 

Operational  Science  and  Technology  Request 

RFI 

Request  for  Information 

SLO 

Scientific  Liaison  Officer 

STSR 

Science  and  Technology  Support  Request 
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Author's  Note 

This  document  records  the  guidance  and  research  approach  adopted  by  DSTO  in  the 
period  2010-2012  in  support  of  campaign  monitoring  and  assessment.  It  was  largely 
written  in  August  2011,  informed  by  workshops  with  DSTO  operational  analysts  and 
campaign  assessment  staff  on  4  July1  and  again  on  24  August2.  It  was  intended  to  capture 
the  logic  behind  the  evaluation  research  approach  that  was  evolving  and  to  provide  a 
generic  guide  for  DSTO  staff  (including  deployed  operations  analysts)  working  in  support 
of  HQJOC,  to  conduct  campaign  assessments  of  Operation  Slipper  in  Afghanistan, 
Operation  Astute  in  East  Timor,  and  Operation  Anode  in  the  Solomon  Islands,  under 
Operational  Science  and  Technology  Support  Request  (OPSTSR)3 132.  Due  to  the  variation 
in  the  form  of  DSTO  support  to  each  of  these  operations,  and  in  particular,  the  termination 
of  embedded  operations  analysts  in  East  Timor,  the  guidance  was  refined  in  September 
2011  to  describe  the  campaign  assessment  support  process  for  Operation  Slipper  only. 

Since  Quarter  3  2011,  this  document  has  been  repeatedly  amended  by  DSTO  staff  to 
become  an  analysis  plan  for  each  quarterly  Operation  Slipper  assessment  cycle,  reflecting 
changes  in  the  process  over  time.  However,  the  version  of  the  guidance  in  this  Technical 
Note  captures  the  original  form  of  the  process  for  Operation  Slipper  as  at  September  2011. 

It  was  circulated  for  comment  among  DSTO  and  HQJOC  staff  at  that  time  and 
incorporates  some  of  the  feedback  received.  Some  minor  amendments  have  also  been 
made  to  reduce  the  security  classification,  and  to  ensure  that  it  can  be  read  and  understood 
as  a  stand-alone  document. 


1  With  thanks  to:  Jamie  Watson,  LTCOL  Brett  Laboo,  Justin  Beck,  David  Graham,  LTCOL  Graham 
Goodwin,  Amelia  Eggerking,  Sharmaine  Ramasamy,  Nathan  Sayers,  Garth  De  Visser,  David 
Matthews,  Rebecca  Karlsson,  Paul  Bennett.  (DSTO) 

2  With  thanks  to:  Simon  Crase,  Jamahl  Bennier,  Tim  McKay,  Nathan  Sayers  (DSTO  Operations 
Support  Centre),  Clive  Walmsley  (HQJOC  Operations  Analysis  Team),  LTCOL  David  Garside 
(HQJOC  Plans,  SOI  J5E),  LTCOL  Scott  Barras  (Deployed  HQ  633  J8),  Edward  Harrington,  Kate 
Cameron,  LTCOL  Brendan  Casey  (Deployed  HQ  633  Operations  Analysis  Team) 

3  Operational  Science  and  Technology  Requests  (OPSTSRs)  are  the  mechanism  through  which 
operationally  urgent  S&T  support  requests  from  the  ADF  are  managed  and  coordinated  by  the 
DSTO  Operations  Support  Centre  (DOSC).  Defence  Science  and  Technology  Organisation,  DSTO- 
GI  01/2011  -  Rev  1,  Operational  Science  and  Technology  Support  Request  (OPSTSR)  Program,  DSTO 
Group  Instruction,  Department  of  Defence,  Canberra,  30  June  2013. 
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1.  Introduction 


1.1  Background 

Recent  Australian  Defence  Force  (ADF)  operations  in  Afghanistan,  East  Timor  and  the 
Solomon  Islands  have  brought  into  focus  the  mechanisms  for  planning  and  evaluating 
large  Defence  activities.  Defence  planners  are  challenged  by  the  requirement  to  translate 
national  strategic  guidance  into  operational-level  plans,  due  to  high  degrees  of  uncertainty 
about  the  interests  of  stakeholders,  likely  impacts  of  courses  of  action,  and  the  timeframes 
for  achievement  of  goals.  In  this  context,  the  ability  to  evaluate  progress  and  identify 
indicators  of  change  is  critical  to  informing  decisions  on  initiating,  maintaining  or  altering 
sequences  of  planned  activity.  In  mid  2010,  DSTO  was  requested  to  provide  support  to  the 
Headquarters  Joint  Operations  Command  (HQJOC)  J5  (Plans)  Branch  to  review  and 
improve  the  process  for  evaluating  operations,  named  Campaign  Assessments.  This 
support  has  been  provided  by  the  Operational  Planning  and  Evaluation  (OPE)  Team, 
within  the  DSTO's  Operational  Support  Centre,  in  collaboration  with  the  DSTO's 
deployed  Operations  Analyst  teams,  and  deployed  Scientific  Liaison  Officers. 

The  problem  to  be  addressed  by  the  OPE  team  was  that  no  formal  guidance  within  Defence 
for  campaign-level  evaluations4.  As  a  consequence.  Campaign  Assessment  for  various 
operations  was  carried  out  using  ad  hoc  formats,  and  without  clear  techniques  or  an  agreed 
framework.  This  lack  of  methodology  resulted  in  inconsistent  information  flows  to  decision 
makers  and  unverified  analysis  leading  to  unreliable  assessments.  As  will  be  detailed  in 
Chapter  2,  the  rigour  and  repeatability  of  the  analytical  processes  is  particularly  important 
to  manage  the  uncertainty  already  present  in  the  complex  Defence  operational  environment. 
For  the  OPE  Team,  this  necessitated  in  the  rapid  development  of  a  scientifically-based 
evaluation  process  to  meet  HQJOC' s  requirements,  which  could  be  immediately  applied  to 
on-going  operations. 


1.2  Scope 

This  report  provides  guidance  for  DSTO  analysts  on  how  to  implement  the  Campaign 
Assessment  process  that  has  been  developed  to  meet  HQJOC's  requirements.  It  was  first 
drafted  for  the  third  quarter  (Q3)  2011  Operation  Slipper  (Afghanistan)  Campaign 
Assessment.  Whilst  it  is  largely  descriptive  of  the  Operation  Slipper  Campaign 
Assessment  process,  it  is  also  intended  as  a  generic  guide  for  assessments  in  other  areas  of 
operations.  It  documents  the  intent,  rationale  for,  roles  and  sequence  of  activities  via  which 
DSTO  provides  scientific  support  for  Campaign  Assessment.  This  report  does  not 
represent  the  body  of  scientific  research  that  underpins  that  support,  as  this  is  presented  in 


4  Acknowledging  that  campaign  assessment  is  covered  or  referred  to  in  a  number  of  ADF  Doctrine 
Publications,  but  these  do  not  provide  a  process  for  executing  assessments  or  conducting  campaign- 
level  analysis  of  progress.  See  Australian  Defence  Doctrine  Publication  5.0,  Joint  Planning 
(Provisional)  (2006  January);  Australian  Defence  Doctrine  Publication  00.4-Operational  Evaluation 
(2007  August);  Australian  Defence  Doctrine  Publication  3.0,  Campaigns  and  Operations  (2012  July). 
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a  number  of  separate  DSTO  reports5.  The  scope  of  this  report  is  to  document  the  process 
and  some  of  the  implications  of  its  design,  to  inform  practitioners  and  to  serve  as  a 
baseline  for  future  research. 

Given  the  high  tempo  and  immediacy  of  the  operational  application  of  this  research, 
formal  review  and  validation  is  recommended  as  the  subject  of  future  research.  This 
research  would  draw  upon  the  findings  of  all  Campaign  Assessment  reports,  on 
measurement  of  client  satisfaction,  and  would  be  assessed  against  theoretical  standards  for 
validity. 


1.3  Aims 

The  aim  is  to  provide  clarity  on  how  DSTO  analysts  can  support  the  Campaign 
Assessment  process  by  defining  the  boundaries  of  DSTO  support,  and  providing  detail  on 
the  DSTO  roles  and  analytical  steps  that  are  required.  The  result  should  be  a  transparent 
and  theoretically  valid  analysis  framework  for  support  to  evaluation  of  ADF  operations. 
As  mentioned  in  Section  1.3,  due  to  the  immediacy  of  application  of  the  CA  process,  the 
achievement  of  these  aims  has  not  been  formally  tested,  and  is  suggested  as  the  subject  for 
future  research. 


2.  Context  of  Campaign  Assessment 
2.1  Definition  of  Campaign  Plans 

As  a  long-term  planning  instrument,  campaign  plans  are  designed  to  articulate  Defence's 
intended  contribution  to  a  Government  initiative,  and  translate  this  intent  into  a  cohesive 
program  of  military  activity  that  will  align  with  Government  strategy  from  start  to  finish 
(Australian_Department_of_Defence  2006).  In  a  Whole  of  Government  context,  as  shown 
in  Figure  1,  a  series  of  National  Objectives  are  derived  from  the  National  Strategy,  and 
these  are  translated  into  diplomatic,  information,  military  and  economic  (DIME) 
objectives.  The  ADF  operational  framework  then  produces  a  campaign  plan  which  breaks 
the  military  strategic  objectives  into  ADF  objectives  and  Effects,  which  can  then  be 
manifested  in  operational  tasking  (Australian_Department_of_Defence  2006). 


5  See  A.  Hickman  and  R.  Karlsson  "Logic  Assumptions  and  Risks  Framework  Applied  to  Defence 
Campaign  Planning  and  Evaluation",  DSTO-TR-2840,  Defence  Science  and  Technology 
Organisation,  2013;  A.  Hickman,  G.  DeVisser  and  R.  Karlsson  "Lead  Indicators  Techniques  Applied 
to  HQJOC  Campaign  Assessment"  DSTO-TN-  Draft,  Defence  Science  and  Technology 
Organisation,  2013. 
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SRATEGIC  ASSESSMENT 


STRATEGIC  ASSESSMENT 


CAMPAIGN  EFFECTIVENESS  ASSESSMENT 


STRATEGIC  FRAMEWORK 


OPERATIONAL  FRAMEWORK 


Figure  1:  Defence  operational  planning  framework  within  a  whole  of  Government  strategic 
planning  structure. 


All  campaign  plans  refer  to  ADF  objectives  as  Campaign  Objectives,  and  each  Objective  is 
addressed  in  the  campaign  plan  with  a  Line  of  Operation  (LoO),  as  shown  in  Figure  2. 
Current  campaign  plans  comprise  between  three  and  six  LoOs.  Under  the  LoOs,  each 
Objective  is  accompanied  by  a  Decisive  Condition  which  describes  what  success  is  expected 
to  look  like,  providing  the  necessary  detail  for  interpretation  of  high  level  guidance  into 
planned  action.  The  Decisive  Condition  is  then  further  broken  down  into  a  set  of  Effects 
(or  progress  milestones),  which  specify  the  means  by  which  the  ADF  expects  to  bring 
about  the  Decisive  Conditions  that  will  achieve  each  Objective.  Each  Effect  describes  a 
condition  or  change  in  condition  that  is  expected  to  be  achieved  by  a  series  of  tasks.  In  this 
way.  Effects  define  the  boundaries  for  the  range  of  activities  that  the  Commander  Joint 
Operations  (CJOPS)  plans  to  undertake  and  scope  of  influence  he  expects  to  have  in  the 
area  of  operations. 
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Figure  2:  Lines  of  Operation  structure  for  campaign  plan. 

The  campaign  plan  can  also  be  thought  of  as  a  bottom-up  process  of  aggregation  where 
the  cumulative  achievement  of  Effects  leads  to  the  emergence  of  the  Decisive  Conditions 
(or  pre-conditions)  for  the  ADF  End  State. 

It  is  apparent  that  there  are  multiple  layers  of  planning  and  categorisation  of  goals 
between  action  on  the  ground  and  assessment  of  whether  overall  strategic  objectives  are 
being  achieved.  The  causal  links  between  the  diverse  and  dynamic  range  of  tasks 
occurring  at  the  tactical  level,  and  operational-level  Objectives  are  often  not  clear  for  a 
number  of  reasons,  including,  but  not  limited  to: 

•  there  may  be  a  lag  between  task  action  and  visible  impact 

•  objectives  are  likely  to  be  achieved  as  a  result  of  the  cumulative  effect  of  a 
combination  of  tasks 

•  the  impact  of  a  task  may  be  interdependent  with  a  number  of  non-operational 
variables  which  are  not  easily  observable 

•  some  Defence  tasks  may  be  aimed  at  supporting  the  achievement  of  the  objectives 
of  other  Australian  Government  agencies 

•  progress  towards  an  Objective  may  be  an  emerging  condition,  which  is  facilitated 
by  Defence  presence,  but  cannot  be  advanced  by  any  specific  task. 

This  lack  of  clarity  and  complex  layering  of  campaign  plans  makes  planning  of  timelines 
and  tracking  of  progress  problematic.  However,  campaign  plans  must  be  carefully 
monitored  to  identify  divergence  from  expected  progress  paths,  or  to  provide  assurance 
that  current  activities  and  resources  are  achieving  expected  results.  This  type  of  visibility, 
referred  to  in  the  ADF  as  situational  awareness  is  critical  for  constant  adaptation  and 
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optimisation  of  effort  in  the  evolving  operating  environment.  In  order  to  achieve  this  type 
of  visibility,  a  detailed  process  is  required  for  monitoring  all  aspects  of  the  campaign 
plan's  execution,  as  well  as  the  environment  in  which  it  is  being  executed.  For  this 
purpose,  a  continuous  process  of  monitoring  and  evaluation  of  all  major  operations  is 
carried  out  by  HQJOC,  referred  to  as  Campaign  Assessment  (CA). 


2.2  Definition  of  Campaign  Assessment 

The  purpose  of  CA  is  to  support  effective  planning  and  decision  making  through 
assessment  of  progress  of  operations  against  the  campaign  plan.  In  the  complex  and 
constantly  evolving  Defence  operational  environment,  monitoring  and  evaluation  enables 
planners  at  all  levels  to: 

•  maintain  visibility  of  whether  the  original  Commander's  intent  is  being  sustained 

•  identify  whether  resources  are  being  put  to  best  effect  (i.e.  whether  gains  are  being 
built  upon,  and  where  less  effective  activities  can  be  discontinued 

•  assess  the  accuracy  and  appropriateness  of  the  campaign  plan  and  where  necessary, 
adapt  it  to  align  with  changes  in  the  environment. 

As  discussed  in  the  previous  section,  the  complexity  and  tempo  of  operations  means  that 
positive  or  negative  trends  are  not  always  clear.  One  of  the  mechanisms  for  overcoming 
this  is  to  focus  on  measurement  of  the  achievement  of  Effects.  This  necessitates  a  detailed 
system  of  monitoring  and  evaluation,  using  a  framework  of  multiple  indicators  of 
progress  for  each  Effect  to  be  monitored  over  time.  These  indicators  are  generally  referred 
to  as  Measures  of  Effect  (MoEs)  (U.S.  Joint  Forces  Command  2010  ),  and  can  be  in  a  variety 
of  forms  such  as  quantitative,  qualitative,  retrospective,  current  and  predictive;  either 
directly  related  or  broadly  contextual.  The  DSTO  evaluation  team  draw  upon  monitoring 
and  evaluation  science  to  assist  military  planners  to  identify  a  robust  and  comprehensive 
set  of  MoEs.  CA  is  then  carried  out  by  monitoring  this  set  of  MoEs  across  regular 
assessment  cycles,  in  combination  with  broader  contextual  monitoring  of  the  operating 
environment,  as  illustrated  in  Figure  3  below.  As  Figure  3  shows,  CA  monitoring  may  also 
draw  upon  tactical-level  internal  task  reporting  by  deployed  forces,  where  Measures  of 
Performance  (MoP)  for  achievement  of  each  task,  may  be  examined  for  overall  positive  or 
negative  trends,  or  evidence  of  persistent  risks  to  success6. 


6  As  discussed  further  in  Section  3.2.1,  MoP  data  may  be  used  in  CA,  as  are  many  other  forms  of 
reporting,  but  they  are  not  linked  to  MoEs  which  are  a  component  of  the  CA  process  structure. 
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Figure  3:  Illustration  of  Inputs  to  Campaign  Assessment,  including  Effect  to  Task  relationship 
derived  from  Campaign  Flans. 


The  emphasis  on  contextual  monitoring  and  achievement  of  Effects,  rather  than  simple 
summation  of  the  success  in  a  collection  of  tasks,  is  a  shift  from  progress  reporting  in  the 
past.  This  is  in  recognition  that  measurement  of  the  success  of  modern  military  operations, 
as  with  any  other  large-scale  program  or  system  interventions,  must  consider  the 
sustainability  of  achievements,  and  the  flow  on  (i.e.  secondary)  effects  of  activities  upon 
the  society  in  which  they  are  conducted.  This  is  particularly  the  case  in  stabilisation 
operations,  where  the  intervening  forces  and  agencies  will  be  held  accountable  for  the 
extent  to  which  the  positive  changes  they  achieve  is  enduring,  and  whether  they  have 
negative  secondary  effects  in  other  sectors  of  society.  Taking  account  of  this  broader 
context  in  CA  requires  the  rapid  synthesis  of  large  amounts  of  data  from  diverse  sources  in 
a  structured  format  that  enables  comparison  and  conclusions  to  be  drawn  across  the  span 
of  the  campaign  plan.  The  challenge  of  expanded  assessment  was  a  motivation  for  HQJOC 
to  enhance  the  CA  process,  and  to  seek  DSTO  support  in  doing  so. 
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A  more  detailed  account  of  how  CA  is  executed,  including  sequencing,  roles  and  reporting 
outputs  is  provided  in  the  remaining  chapters  of  this  report.  Prior  to  describing  the 
process  developed  by  the  DSTO  however,  it  is  necessary  to  discuss  the  practical 
constraints  and  considerations  of  the  research  context  that  have  shaped  the  design  of  the 
CA  process. 


2.3  Practical  constraints  and  considerations 

Support  from  the  DSTO  was  requested  by  HQJOC7  under  OPSTSR  132,  to  improve  the 
structure  and  analysis  content  of  the  CA  process.  Specifically,  DSTO  was  requested  to: 

a.  revise  the  structure  of  the  CA  framework  under  which  information  needs  are 
identified,  and  data  is  subsequently  analysed 

b.  identify  data  gathering  and  analysis  tools  and  techniques  that  improve  the  validity  and 
accuracy  of  measures  of  progress 

c.  support  the  revision  of  Campaign  plans,  effects,  metrics  and  indicators  to  adapt  to  the 
evolution  of  operations  and  the  information  requirements  of  decision  makers; 

d.  provide  social  and  political  analysis  expertise  to  support  the  expanded  scope  of 
monitoring  and  evaluation 

e.  establish  a  team  with  an  average  of  two  staff  available  for  each  Campaign  assessment 
cycle  with  country  expertise  in  each  of  the  three  current  major  areas  of  operation: 
namely  Afghanistan,  East  Timor  and  the  Solomon  Islands. 

Provision  of  these  forms  of  support  is  constrained  by  a  number  of  characteristics  of  the 
context  of  application,  and  a  number  of  additional  constraints  and  considerations  have 
emerged  during  the  course  of  research  on  potential  approaches  and  techniques.  Each  of 
these  are  discussed  below. 

2.3.1  Operational  Time  Constraints 

Attempting  to  enhance  the  CA  process  in  the  context  of  high  operational  tempo  imposes  a 
number  of  constraints.  The  aim  of  the  CA  improvement  is  to  reach  a  compromise  between 
what  was  achievable  with  the  time  and  information  available  in  the  area  of  operations  and 
the  sort  of  situational  awareness  required  to  enable  CJOPs  to  make  informed  decisions 
about  the  current  and  likely  future  direction  of  the  campaign.  This  means  refinement  and 
iterative  improvements  which  can  be  made  without  disrupting  the  on-going  assessment 
cycles,  regardless  of  what  DSTO  analysts  might  consider  to  be  scientific  best  practice. 

Implementing  improvements  to  CA  during  on-going  operations  is  constrained  by  a  lack  of 
staff  availability  for  CA  data  collection  and  analysis,  which  precludes  the  invention  of  a 
new  process,  or  even  revolutionary  changes  to  the  existing  one.  The  techniques 


7  Classified  HQJOC  Minute,  dated  31  May  2010.  The  initial  request  for  support  was  later  formalised 
under  Operational  Science  and  Technology  Support  Request  (OPSTSR)  132. 
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incorporated  in  the  CA  process  must  be  selected  with  the  aim  of  minimising  the  impact  of 
data  collection  and  analysis  upon  normal  operational  activities.  As  a  result,  any  practical 
enhancement  of  the  CA  process  must  involve  a  minimum  of  staff  time  investment.  This 
applies  to  staff  resources  in  the  HQJOC  J5  (Plans)  Branch,  and  also  to  the  time  and  staff 
available  in  the  respective  deployed  ADF  Task  Force  Headquarters  (HQ).  Another 
implication  is  that  there  is  no  staff  time  available  for  training  or  familiarisation  on  a  new 
CA  process,  so  any  changes  to  monitoring  and  evaluation  activities  or  roles  needs  to  fit 
easily  into  established  structures  in  both  HQJOC  and  the  deployed  HQs. 

Finally,  there  are  minimal  resources  available  to  dedicate  to  analysis  of  expanded  data 
sources.  Therefore,  the  CA  process  can  not  be  improved  by  simply  increasing  the  number 
of  analysts,  and  it  is  apparent  that  a  solution  has  to  be  found  for  more  efficient  processing 
of  an  expanded  data  set,  either  through  analysis  techniques  or  information  management 
tools. 

2.3.2  Adaptability 

HQJOC's  requirement  to  develop  a  single,  consistent  CA  approach  that  can  be  applied  to 
different  campaigns,  necessitates  an  approach  that  is  general  enough  to  be  adaptable  to 
various  forms  of  operations,  but  also  specific  enough  to  define  and  capture  information  on 
all  aspects  of  each  Campaign.  This  requires  a  structured  framework  that  is  clearly  relevant 
to  ADF  mission  and  priorities,  but  includes  options  for  adapting  the  evaluation  focus  to 
suit  the  operational  context.  In  particular,  opportunities  to  create  culturally  appropriate 
MoEs  are  critical,  so  that  assessments  can  be  made  on  what  progress  is  expected  to  look 
like  in  different  populations  with  diverse  baselines  of  capacity  and  differing  expectations 
of  authority  and  well  being. 

This  adaptability  extends  to  a  requirement  for  the  CA  process  to  maintain  ADF 
operational  intent,  but  also  be  able  to  be  adjusted  to  account  for  broader  coalition/UN 
mission  assessment  standards  and  planning  where  relevant. 

2.3.3  Evaluation  in  a  non-permissive  environment 

A  third  constraint  is  the  likelihood  of  gathering  data  for  CA  in  an  environment  which  is 
subject  to  conflict  or  crisis;  referred  to  as  a  non-permissive  environment  by  the  ADF.  The 
likelihood  of  operating  in  a  non-permissive  environment  significantly  constrains  the 
practicality  of  gathering  evidence  on  progress  directly  from  the  target  population.  When 
operating  in  environments  where  there  may  be  adversaries,  or  where  there  is  potential  for 
the  situation  to  deteriorate  if  some  forms  of  information  are  made  available,  the  scope  for 
consultation  is  limited.  This  is  contrary  to  a  large  portion  of  existing  techniques  for 
enhancing  evaluation,  which  advocate  participative  research  (Van  der  Riet  2001)  and 
collaborative  evaluation  wherever  possible.  However,  while  the  merits  of  high  degrees  of 
consultation  and  local  participation  are  acknowledged,  this  requires  longer  timeframes 
and  levels  of  engagement  than  are  usually  viable  for  military  operations  in  post-crisis  or 
conflict  environments. 
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3.  Theoretical  Framework 


The  CA  process  described  in  this  document  is  based  on  Funnell  and  Rogers  (2011)  Program 
Theory  evaluation  structure  which  will  be  defined  and  reconsidered  here  in  the  Defence 
context.  As  this  report  is  aimed  at  documenting  the  process  for  DSTO  support  to  CA,  and 
not  the  science  behind  its  development,  the  historical  and  theoretical  detail  on  Program 
Theory  is  contained  elsewhere  in  another  DSTO  report  (Anderson  2004;  Hickman  2013). 
The  theoretical  framework  will  be  explained  here,  only  to  establish  an  understanding  of  the 
rationale  for  the  structure  and  sequencing  of  the  CA  process. 


A  Program  Theory-based  evaluation  framework  requires  that  evaluations  begin  with  clear 
statements  of  constraints,  scope  and  requirements  (Weiss  1972),  referred  to  in  Figure  4  as 
the  Context  for  the  Evaluation.  Within  the  framework,  this  clear  context  can  then  be 
translated  into  a  Monitoring  Plan  and  an  Evaluation  Plan,  which  have  clear  purpose,  scope 
and  authority. 


Background  and  theory  to 
task 


(History/  stakeholders  / 
scope  of  evaluation  task) 


Context  of  Program  to 
be  evaluated 

(Considerations, 
constraints  of  Campaign 
and  client  requirements) 


For  existing  operations,  this 
will  be  covered  by  the  extant 
Campaign  Plan  and  MoE  set. 


Monitoring  Plan 
(What  are  we  doing  and 
what  do  we  need  to  know 
for  accountability?) 


y  Evaluation  Plan 
\  (Consensus  on 
I  frequency,  resources 
I  and  scale) 


- X - 

Each  comprising: 


Data 

Collection 

Strategy 


Analysis 

Strategy 


Reporting 

Strategy 


Implementation 

Strategy 


Figure  4:  Structure  of  Program  Theory  Evaluation  Framework  for  DOSC  support  to  Campaign 
Assessments 
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In  a  conventional  Program  Theory-based  evaluation  framework.  Program  Logic  would 
ordinarily  be  included  in  the  Analysis  Plan  and  would  involve  significant  engagement 
with  the  commissioner  of  the  assessment  at  the  outset  (i.e.  HQJOC).  The  timeframes  for 
assessments  to  date  have  not  allowed  for  the  CA  process  to  include  scoping  of  program 
logic  from  first  principles,  or  even  the  full  exploration  of  the  internal  logic  of  existing 
campaign  plans.  Instead,  the  extant  campaign  plans  and  the  Effects  defined  within  them 
were  used  as  the  basis  of  Program  Logic  for  the  development  of  the  CA  evaluation 
framework. 

As  Weiss  (1997)  emphasised  in  the  theory  of  evaluation  science,  it  is  essential  that 
evaluators  establish  at  the  outset  how  the  results  of  the  assessment  are  expected  to  be  used 
by  decision  makers,  how  they  understand  the  assessment  to  be  structured  and  the  role 
they  expect  to  play  in  the  process.  In  future,  CA  cycles  would  ideally  commence  with 
DOSC  staff  discussions  with  HQJOC  to  establish  a  comprehensive  shared  understanding 
of  the  logic  within  extant  campaign  plans  and  this  would  combine  with  the  Context  for 
Evaluation  (discussed  in  Section  2.2  above)  to  define  the  scope  of  CJOPS'  information 
requirements.  This  consultation  with  key  stakeholders  to  establish  consensus  on  logic,  is 
referred  to  by  Bickman  (1987)  as  a  'Program  Logic  model',  which  becomes  the  baseline  for 
the  evaluation.  The  importance  of  taking  the  time  to  establish  a  consensus  is  reiterated  by 
Kilcullen  (2009)  as  follows: 

Because  we  need  to  track  so  many  things  for  so  many  people,  a  shared  diagnosis  -  a 
vision  of  what  the  conflict  is,  and  what  is  driving  it  -  is  essential.  Neglecting  this 
diagnosis  risks  a  situation  where  analytical  staff  are  drowning  in  data  -  lacking  a 
clear  conception  of  what  matters  and  what  does  not,  they  collect  on  everything, 
creating  a  mass  of  disparate  data  that  makes  tracking  progress  harder. 

This  reinforces  the  need  for  the  client  commissioning  an  evaluation  to  accept  that  they  will 
be  required  to  invest  time  in  the  development  of  the  evaluation  process,  and  not  just  wait 
to  be  recipients  of  the  product.  This  enables  evaluators  to  support  the  assessment  with  a 
clear  understanding  of  the  concepts  of  progress  that  they  are  making  assessments  of,  and 
can  also  be  used  as  guidance  on  what  will  be  relevant  for  those  required  to  provide  data. 


3.1  Evaluation  Questions 

Evaluation  Questions  are  arguably  the  most  important  component  of  the  Program  Theory 
evaluation  framework,  because  they  determine  what  information  the  CA  will  be  based 
upon,  and  should  also  form  the  basis  of  the  Analysis  Strategy  (see  lower  layer  of 
framework  in  Figure  4)  of  both  the  Monitoring  and  Evaluation  Plans.  It  is  the  analysis 
prompted  by  the  Evaluation  Questions  that  has  greatest  impact  upon  the  quality  of  a  CA. 
Within  the  CA  construct.  Evaluation  Questions  are  represented  by  MoEs,  as  they  are  the 
mechanism  through  which  campaign  plan  Effects  are  broken  down  into  more  detailed 
information  requirements  to  determine  the  scope  of  data  gathering  and  analysis.  In  some 
instances,  additional  Evaluation  Questions  may  be  included,  where  there  are  identified 
information  requirements  that  are  not  covered  under  Effects  by  MoEs,  such  as  broader 
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environmental  context.  As  explained  with  reference  to  Program  Logic  above.  Evaluation 
Questions  would  ordinarily  be  derived  from  extensive  consultation  with  HQJOC  to 
explore  the  Effects  defined  in  the  campaign  plan.  Ideally,  an  MoE  set  and  any  additional 
evaluation  questions  for  each  Effect  would  be  derived  from  this  process.  However,  DSTO 
has  been  asked  to  provide  support  to  CA  after  Effects  and  MoEs  are  already  in  place  for 
each  current  operation.  Therefore,  this  step  of  the  framework  occurs  as  a  review  and 
refinement  of  MoEs,  rather  than  devising  evaluation  questions  from  genuine  exploration 
of  the  Program  Logic. 

Nonetheless,  there  are  brief  intervals  for  refinement  of  Evaluation  Questions  through 
review  of  existing  MoE  sets  for  each  operation.  When  requested  on  an  ad  hoc  basis,  the 
DSTO  team  works  with  HQJOC  and  sometimes  deployed  HQ  staff,  to  reconsider  the 
comprehensiveness,  cohesiveness  and  appropriateness  of  the  MoEs.  This  is  intended  to 
include  a  refinement  of  the  underpinning  criteria,  by  which  each  MoE  will  be  rated,  and 
accompanying  guidance  notes,  which  will  advise  respondents  on  how  to  interpret  each 
MoE,  and  how  they  should  scope  their  responses. 

Although  refinement  of  the  MoE  set  is  an  important  step  in  the  CA  cycle,  metrics  are  not 
changed  from  cycle  to  cycle  unless  they  are  outdated  or  no  longer  appropriate  for  the 
evolving  campaign.  The  minimisation  of  change  to  Evaluation  Questions  (i.e.  MoE)  is 
aimed  at  maintaining  as  much  as  possible  a  consistent  framework  of  measurement  of 
progress  over  time  so  that  stakeholders  can  become  accustomed  to  collecting  certain  forms 
of  data  to  contribute  to  CA,  and  decision  makers  can  expect  a  reliable  and  consistent 
source  of  information.  Most  importantly,  minimising  change  in  Evaluation  Questions 
facilitates  the  establishment  and  maintenance  of  a  baseline  of  information  against  which 
comparisons  can  be  made  over  time. 


3.2  Indicators  and  observable  trends 

The  terms  'indicators'  and  'observable  trends'  are  frequently  used  in  the  CA  context,  and 
there  are  a  number  of  theoretical  implications  related  to  them  that  are  worth  noting  before 
proceeding  to  describe  the  process  in  which  they  are  used.  For  our  purposes  they  are 
defined  as  quantitatively  or  qualitatively  observable  variables  that  provide  evidence  of 
changes  in  a  situation  or  factor  we  are  monitoring  (Etheridge  2002). 

CAs  evaluate  progress  made  in  the  preceding  quarter  (i.e.  Tag  indicators')  as  well  as 
likelihood  of  achieving  future  milestones,  to  identify  likely  risks  and  provide  cues  for  re- 
evaluation  of  aspects  of  the  plan  such  as  timeframes  or  force  structure.  In  this  way,  an 
effective  CA  process  will  be  instrumental  in  keeping  the  campaign  plan  up  to  date  and 
relevant,  by  providing  the  opportunity  to  align  endorsed  planning  with  the  reality  on  the 
ground,  and  vice  versa.  The  challenge  here  is  that  assessments  of  future  progress  can  only 
be  based  on  what  is  visible  in  the  present.  Therefore,  when  gathering  data,  it  is  worth 
considering  that  some  indicators  may  not  seem  important  now,  but  may  be  highly  valued  in 
combination  with  other  indicators  as  projections  of  potential  success  or  failure.  Examples  of 
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these  sort  of  indicators  are  Afghan  National  Army  graduation  numbers,  local  agricultural 
prosperity,  observable  growth  in  new  enterprises  or  enrolments  in  schools.8 

Therefore,  indicators  are  signals  of  change  -  they  are  not  in  themselves  the  desired  end 
state  we  are  trying  to  implement.  For  example,  we  may  monitor  the  price  of  daily 
foodstuffs  as  a  means  of  measuring  the  levels  of  stability  in  society.  Therefore,  a  steadying 
in  prices  is  a  possible  indicator  of  greater  societal  stability,  but  we  must  remember  that 
normalisation  of  prices  is  not  our  ultimate  objective  -  societal  stabilisation  is.  These 
indicators  are  referred  to  variously  in  the  literature  as  'proxy  measures',  'lead  indicators' 
(Hickman  2013a)  or  as  Kilcullen  (2009)  defines  them: 

assessment  staffs  are  looking  for  surrogate  indicators  that  allow  them  to  detect 
deeper  trends  in  the  environment  that  may  not  be  directly  observable.  They  are  also 
looking  for  clusters  -  indicators  that  tend  to  occur  together  and,  taken  in  context,  can 
be  interpreted  together  to  generate  a  picture  of  overall  trends. 

This  understanding  of  the  types  and  implications  of  indicators  is  informative  for  efficient 
and  effective  data  gathering  and  analysis  throughout  the  CA  process,  defined  in  the  next 
chapter. 

3.2.1  Measures  of  Performance  (MoPs) 

As  mentioned  in  Section  2.2,  MoPs  are  sometimes  drawn  upon  as  a  data  source  in  CA,  but 
unlike  MoEs,  they  are  not  a  component  of  the  CA  process.  This  is  counter  to  a  common 
misconception  that  the  sum  of  MoP  ratings  can  be  quantified  as  an  indicator  of  the 
achievement  of  higher-level  MoEs.  This  misconception  is  apparently  derived  from  the  use  of 
MoPs  in  financial  and  commercial  organisational  effectiveness  frameworks  such  as  the 
"balanced  scorecard"9.  This  confusion  of  terminology  has  been  observed  during  briefings  of 
ADF  members  on  the  CA  process,  when  there  is  an  expectation  that  the  MoPs  used  to 
evaluate  tasks  at  or  the  performance  of  staff  at  the  tactical  level,  are  directly  linked  to  the 
MoEs  used  in  CA.  This  is  not  the  case,  as  is  frequently  clarified  in  evaluation  theory 
literature  and  military  guidance10,  which  asserts  that: 

MOPs  are  closely  associated  with  task  accomplishment,  whereas  MOEs  measure  the 
attainment  of  an  end-state,  achievement  or  objective.  Consider  MOPs  and  MOEs  in 
terms  of  task  and  purpose.  MOPs  relate  to  accomplishment  of  the  task,  whereas 
MOE's  relate  to  the  accomplishment  of  the  purpose. (U.S.  Joint  Forces  Command 
2010 ) 


8  In  "Power  vs  Force:  The  Hidden  Determinants  of  Human  Behaviour",  David  Hawkins  refers  to 
this  as  making  inaccurate  'presumptions  about  deterministic  linear  sequences',  which  only  focuses 
on  direct  and  obvious  causal  variables,  and  fails  to  recognise,  'unobservable',  or  less  direct 
influencing  factors,  (quoted  in  Meharg  2009) 

9  For  example  see  "The  Balanced  Scorecard  Institute", 

http:/ /www.balancedscorecard.org/bscresources/ performancemeasurement/tabid/59/default.as 
px. 

10  See  also,  "NATO  Operations  Assessment  Handbook",  January  2011;  and  S.  Meharg,  "Measuring 
What  Matters  in  peace  Operations  and  Crisis  Management"  2009. 
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Some  MoPs  have  no  relation  to  Effects  at  all,  and  relate  only  to  completing  routine 
operational  functions  that  are  separate  from  the  achievement  of  an  Effect.  For  tasks  that  are 
related  to  the  achievement  of  Effects,  MoPs  only  tell  us  that  those  tasks  have  been 
completed,  or  may  document  the  obstacles  that  prevented  the  task  from  being  completed. 
This  does  not  indicate  whether  progress  is  being  made  towards  an  overall  Effect,  but  it  may 
contribute  to  an  explanation  of  why  progress  is,  or  isn't  being  made.  For  example,  the  MoPs 
for  a  range  of  tasks  may  provide  consistent  evidence  that  current  force  structure  or 
resourcing  is  inadequate  to  enable  the  deployed  force  to  achieve  planned  Effects. 


3.3  Variety  of  information  inputs 

When  collecting  information  as  indicators  for  MoEs,  monitoring  and  evaluation  theories 
emphasise  the  importance  of  drawing  upon  numerous  diverse  sources  (Donaldson  2009). 
An  assessment  on  an  MoE  has  little  validity  if  it  is  based  upon  only  a  single  evidence 
source.  Instead,  it  is  preferable  to  cross-validate  indicators  with  data  from  alternate 
sources  and  to  take  into  consideration  other  factors  such  as: 

•  the  situational  context 

•  examples  of  change  with  comparable  sample  groups/  seasons/  timeframes 

•  significant  events  during  the  reporting  period 

•  aggregation  of  the  Measures  of  Performance  assigned  to  any  related  tasks 

•  data  reliability  and  the  motivation/bias  of  interviewees  or  generators  of  the  data. 

Therefore,  when  sourcing  inputs  for  MoE  indicators  it  is  necessary  to  look  broadly  and  try 
to  avoid  assumptions  of  oversimplified  linear  causality  (i.e.  thinking  that  a  particular 
action  is  the  only  influencing  factor  for  the  achievement  of  the  desired  result). 


3.4  Assumptions 

A  central  tenant  of  the  Program  Evaluation  Theory  applied  here  is  that  identification  of 
accurate  indicators  is  contingent  upon  understanding  how  planners  assume  progress  is 
going  to  be  achieved,  and  what  it  is  assumed  to  look  like.  Without  surfacing  these 
assumptions,  and  establishing  some  sort  of  consensus  on  them,  indicators  of  progress  will 
be  highly  subjective,  and  their  relevance  will  not  be  understood  by  anyone  with  a  different 
vision  of  progress.  This  is  widely  recognised  in  evaluation  literature  and  was  articulated 
by  the  UK  Ministry  of  Defence's  Development,  Concepts  and  Doctrine  Centre  (Couzens 
2010): 

It  is  only  when  ideas  have  been  put  into  words  that  flaws  in  logic  become  apparent 
and  when  misunderstandings  between  planning  team  members  or  the  planning 
staff,  commanders  and  the  rest  of  the  staff  are  revealed. 


UNCLASSIFIED 


13 


UNCLASSIFIED 

DSTO-TN-1233 

When  conducting  CA  we  must  make  sure  that  the  implicit  assumptions  within  the 
campaign  plan  about  achievements  of  Effects  are  made  explicit.  A  shared  vision  of  success 
requires  the  surfacing  of  the  assumptions  that  underpin  the  theory  of  how  change  is 
expected  to  occur. 

This  is  particularly  important  when  applying  a  plan  over  long  periods  through  staff 
rotations.  Also,  where  there  are  multiple  agencies,  departments  or  organizations  involved, 
each  will  have  their  own  logic,  vocabulary  and  vision  of  how  progress  can  best  be 
achieved.  These  differing  perspectives  and  vocabularies  make  the  articulation  of 
assumptions  critical  to  establishing  a  shared  understanding  of  how  objectives  will  be 
achieved. 

It  can  be  concluded  that  incorporation  of  some  form  of  mechanism  for  surfacing  and 
monitoring  assumptions  is  necessary  in  an  evaluation  framework.  To  date,  inclusion  of  a 
specific  assumptions  component  to  the  CA  process  has  not  been  achieved  within  the 
constraints  outlined  in  Section  2.3.  However,  more  recently,  the  DSTO  team  has  developed 
a  Logic,  Assumptions  and  Risks  Framework  (LARF)  (Hickman  2013)  designed  to  ensure 
that  this  important  component  of  evaluation  is  integrated  into  the  CA  process.  The  LARF 
has  been  trialled  on  a  variety  of  campaign  planning  and  CA  phases,  and  may  be 
formalised  as  part  of  CA  in  the  future. 


4.  Method  for  Evaluation  Support  to  Campaign 

Assessment 


The  CA  context  and  the  theoretical  understanding  outlined  in  the  preceding  chapters 
underpins  the  evaluation  method  that  has  been  developed  for  DSTO  provision  of  support 
to  CA.  DSTO  support  to  CA  can  be  summarised  as  providing  the  following: 

•  analysis  of  data  that  underpins  HQJOC-led  CAs  reports  to  CJOPS  and  the  Chief  of 
the  Defence  Force 

•  advice  and  support  to  deployed  DSTO  Operations  Analysis  Teams  to  coordinate 
in-theatre  responses  against  Campaign  MoEs 

•  planning  and  execution  of  Fly  Away  Teams  for  CA  data  gathering,  as  required  in 
various  areas  of  operation 

•  on-going  research  to  improve  the  CA  process  and  to  contribute  to  activities 
pertinent  to  Campaign  planning  and  assessment  in  order  to  improve  ADF 
capability  to  train,  deploy,  execute,  adapt  and  complete  operations. 

The  CA  method  detailed  here  enables  the  first  3  forms  of  support,  and  is  an  outcome  of  the 
fourth. 
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4.1  Monitoring  and  Evaluation  Plans 

In  the  past,  the  entire  process  for  an  assessment  cycle  has  been  carried  out  in 
approximately  16  weeks,  as  shown  in  an  example  timeline  at  Appendix  A,  which  depicts  a 
CA  process  carried  out  using  a  deployed  Operations  Analysis  (OA)  Team.  However, 
repeated  iterations  of  this  process  have  clearly  demonstrated  that  this  timeframe  is 
dependent  on  unsustainable  staff  levels  and  significantly  compromises  the  quality  of  data 
collection  and  analysis  and  does  not  allow  for  sufficient  validation  of  findings  and  optimal 
report  writing.  It  has  been  concluded  by  the  CA  Science  Team  Leader  that  a  minimum  of 
24  weeks  for  each  cycle  is  required  for  a  consistent  quality  CA.  Allowing  any  less  time 
than  this  will  need  to  be  negotiated  between  HQJOC  and  DSTO  to  agree  on  how  minimum 
scientific  standards  can  be  maintained  and  which  components  of  the  process  will  be 
abbreviated  or  circumvented. 

In  order  to  draw  out  the  different  tasks,  information  domains  and  responsibilities 
contained  in  each  CA  cycle,  a  distinction  has  been  drawn  between  the  Monitoring  and 
Evaluation  phases  of  the  process.  As  summarised  in  Figure  5,  for  a  conventional  CA  of  a 
large-scale  overseas  operation,  the  deployed  HQ  will  be  largely  responsible  for  the 
monitoring  of  ADF  and  coalition  operational  activities,  as  well  as  contribution  to 
monitoring  data  on  the  broader  operational  environment.  This  is  complimented  by  DSTO 
evaluation  team  monitoring  of  indicators  of  operational  progress  and  of  the  operating 
environment,  including  the  activities  and  interests  of  OGAs  and  coalition  partners. 
Evaluation  of  the  monitoring  data  is  then  carried  out  by  the  DSTO  evaluation  team  in 
consultation  with  HQJOC  J5  staff.  This  is  followed  by  a  synthesis  of  trends,  conclusions 
and  recommendations  that  are  captured  collaboratively  by  DSTO  and  HQJOC  staff,  in  a 
CA  report  which  is  circulated  for  comment  to  the  deployed  HQ  and  OGAs,  before  being 
finalised  and  briefed  to  CJOPS. 
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Figure  5:  Campaign  Assessment  Monitoring  and  Evaluation  Roles 


As  shown  in  Figure  5,  the  roles  of  each  of  the  three  contributors  shift  throughout  the 
process,  making  it  very  much  a  collaborative  effort,  which  no  single  contributor  can 
achieve  alone. 

Description  of  the  CA  support  process  in  detail  will  now  be  given,  beginning  with  steps 
and  guidance  on  the  Monitoring  Plan,  followed  by  the  Evaluation  Plan. 


4.2  Monitoring  Plan 

As  part  of  the  CA  process,  the  relevant  deployed  F1Q  is  required  to  provide  the  F1QJOC  J5 
(Plans)  Branch  with  a  submission  of  data  on  progress  made  during  the  reporting  cycle, 
against  each  of  the  Effects  identified  in  the  campaign  plan.  This  submission  is  required  in 
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the  form  of  responses  to  a  series  of  metrics  identified  for  each  effect,  namely  the  MoEs 
discussed  under  3.1  above. 

As  Figure  5  shows,  monitoring  is  largely  at  the  deployed  operational  level  and  therefore 
sits  within  the  responsibility  of  the  deployed  HQ  through  a  deployed  J8,  OA  Team  or 
Scientific  Liaison  Officer  (SLO)  (assisted  by  a  DSTO  fly-away-team).  This  monitoring  then 
transitions  into  the  evaluation  space,  when  the  deployed  HQ  passes  their  monitoring  data 
to  OPE  Team  (via  HQJOC  J5  Branch). 

4.2.1  Data  collection  Strategy 

4.2.1.1  Data  sources 

It  has  been  recognised  that  OA  teams  do  not  always  have  an  organic  collection  capability, 
so  their  inputs  to  the  CA  process  may  be  drawn  from  mining  of  secondary  data  sources, 
rather  than  concentrating  on  gathering  new  and  novel  data.  As  discussed  in  Section  3.2.1 
and  3.3,  data  is  sought  in  the  form  of: 

a.  evidence  for  positive  or  negative  trends  in  each  MoE  (drawn  from  observable 
indicators  of  incidents,  briefs  and  reports,  interviews,  and  aggregated  data  from 
MoP  ratings) 

b.  contextual  insights  into  the  broader  operational  environment  (drawn  from  all 
forms  of  classified  and  open  source  reporting  and  interviews). 

Previous  CA  cycles  have  resulted  in  the  production  of  various  forms  of  data  collection 
plans  in  each  area  of  operation,  which  should  be  available  as  a  starting  point  checklist  of 
primary  sources  of  data  and  responsibilities  for  data  collection. 

4.2.1.2  Socialisation  of  metrics 

Data  collection  is  greatly  assisted  if  respective  deployed  HQ  cells  are  familiar  with  the  CA 
process  and  the  MoEs  that  are  likely  to  relate  to  them.  The  MoE  set  is  supplied  to  OA 
Teams  and  SLOs  in  the  form  of  a  template,  which  includes  guidance  notes  on  how  to 
interpret  each  one,  the  primary  source  documents/ reports  with  which  each  MoE  is 
aligned,  the  scope  of  information  sought,  and  any  associated  points  of  relevance  in 
previous  studies  or  from  other  sources.  The  template  can  also  include  space  for 
respondents  to  report  the  source  of  their  information,  rate  its  validity,  the  sample  size 
used,  timeframe,  comparison  with  previous  reports,  to  note  any  contrary  indicators,  and 
to  give  narrative  examples.  Where  possible  it  is  intended  that  this  MoE  template  is 
socialised  regularly  with  the  members  of  the  deployed  HQ  who  are  responsible  for  any 
form  of  reporting. 

As  part  of  the  socialisation  of  metrics,  nominated  responsibility  for  data  and  sources  will 
need  to  be  reviewed  by  the  OA  Team  each  cycle,  to  take  account  of  new  sources,  changes 
in  reporting  roles  and  lessons  from  previous  cycles.  This  is  a  dynamic  and  ongoing  process 
where  the  regular  briefing  of  deployed  HQ  staff,  recording  of  information  sources  and 
feedback  to  the  OPE  Team  on  the  availability  (or  lack  of)  data  will: 
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•  enhance  understanding  of  the  intent  of  the  process  (ie.  the  impact  of  assessments 
and  the  fact  that  they  are  not  just  compliance  reporting) 

•  pre-empt  any  obstacles  to  data  collection  before  they  become  the  cause  of  'nil 
returns' 

•  provide  an  opportunity  for  communication  of  any  queries  or  corrections  to  HQ 
JOC  before  time  and  effort  is  spent  on  data  collection. 

4.2.1.3  Data  collation 

Collation  of  data  relevant  to  each  MoE  should  be  occurring  within  the  OA  Team  from  the 
day  that  the  previous  cycle  ends.  In  the  past  this  has  been  in  the  form  of  storing  data  under 
MoE-specific  files  in  a  repository  which  is  shared  by  OPE  Team  on  the  DOSC  Sharepoint 
system,  which  has  an  identical  structure  on  both  the  SECRET  and  RESTRICTED  networks. 
It  is  intended  that  this  repository  can  be  added  to  by  anyone  throughout  the  cycle,  either 
copying  or  attaching  data  directly  into  the  appropriate  folders,  or  summarising  for  future 
reference.  It  is  critical  that  all  material  is  accurately  referenced  to  enable  further  review. 

Several  weeks  prior  to  the  agreed  deployed  HQ  submission  date,  deployed  HQ  data 
should  be  collated  by  the  OA  Team  or  SLO,  to  facilitate  the  commencement  of  an  initial 
analysis  phase. 

4.2.2  Analysis  Strategy 

The  OA  Team  or  SLO  (assisted  by  the  CA  fly-away-team)  are  relied  upon  to  synthesise  the 
data  they  have  collated  and  conduct  initial  analysis  of  trends  and  observations  relevant  to 
the  MoEs.  Wherever  possible  these  trends  and  assessments  should  be  drawn  out  of 
discussion  of  the  data  with  deployed  HQ  subject  matter  experts,  reflecting  the  perspective 
of  key  positions  in  theatre. 

There  are  a  number  of  key  analysis  questions  that  should  ideally  be  reflected  upon  when 
synthesising  data  during  this  phase,  which  are  aimed  at  drawing  out  the  insights  that  the 
deployed  HQ  may  be  best  placed  to  elaborate  on.  For  example: 

•  What  tangible  evidence  is  there  of  progress  ratings  for  any  of  the  MoEs?  Are  there 
any  aspects  of  this  evidence  that  would  compromise  its  validity? 

•  What  are  the  factors  currently  inhibiting,  or  likely  to  inhibit  progress? 

•  Where  trends  have  been  indentified,  what  are  the  risks  to  continuing  or  reversing 
them? 

•  Do  any  of  the  assessments/ evidence  provided  by  deployed  HQ  staff  contradict 
previous  assessments?  If  so  can  the  source  area  provide  explanation? 

•  What  broad  narratives  on  progress  and  ultimate  campaign  success  are  apparent  in 
the  area  of  operations? 
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To  the  extent  that  time  permits,  comparative  charting  of  MoE  ratings  and  any  quantitative 
key  indicators,  against  data  from  previous  cycles,  is  highly  valued  as  part  of  the  deployed 
HQ  submission. 

Even  at  this  stage,  where  there  is  still  confusion  over  rating  scales  and  metrics  associated 
with  each  MoE,  queries  should  be  directed  back  to  the  OPE  Team,  so  that  there  can  be 
some  consensus  on  the  basis  for  ratings.  In  some  instances,  greater  clarity  on  the  intent  of  a 
MoE  or  associated  criteria  may  be  given  through  examples  or  additional  indicators. 

4.2.3  Reporting  Strategy 

Following  the  analysis  phase,  MoE  assessments,  evidence,  trends,  and  observations  are 
combined  in  a  draft  submission  which  is  then  workshopped  with  all  deployed  HQ  staff  to 
ensure  accurate  representation  (1  week  before  submission  date).  The  submission  can  then 
be  forwarded  to  the  HQJOC  J5  (Plans)  Branch  and  DSTO  OPE  Team.  This  submission  does 
not  need  to  be  a  polished  report  format  and  can  take  any  form  that  best  suits  the  data 
being  presented.  It  is  important,  however  that  the  submission  has  several  characteristics: 

•  Clearly  identifiable  ratings  and  explanations  for  each  MoE  (including  explanations 
where  no  data  was  available,  or  where  no  assessment  is  offered). 

•  All  assertions  -  whether  contextual  observations  or  MoE  ratings  -  are  supported  by 
examples,  and  explanation  of  potential  causality.  If  there  are  no  indications  of 
causality,  the  lack  of  visibility  is  also  reportable. 

•  Comprehensive  and  consistent  referencing  for  data  to  enable  verification,  to  avoid 
duplication,  and  to  enable  maintenance  of  corporate  knowledge. 

The  use  of  previous  deployed  HQ  CA  submissions  as  a  template,  is  recommended,  after 
first  checking  with  HQJOC  or  the  DSTO  team  that  the  particular  cycle  that  is  being  used  is 
an  example  that  we  would  seek  to  replicate.  If  in  doubt,  submission  of  early  drafts  to  OPE 
Team  will  provide  an  opportunity  for  guidance  on  desirable  content  and  format. 

The  level  of  approval  required  for  the  submission  prior  to  sending  to  HQ  JOC  has  changed 
over  time.  There  have  been  variations  between  a  requirement  for  full  briefing  and 
approval  by  deployed  HQ  prior  to  submission,  to  submission  of  raw  data  from  the 
deployed  HQ  with  no  requirement  for  approval  beyond  the  OA  team.  This  is  an  aspect  of 
the  process  that  will  need  to  be  re-confirmed  each  cycle  to  ensure  that  Commanders  and 
HQJOC  are  aligned  and  that  there  is  ample  time  allowed  for  any  approval  process. 

Following  the  deployed  HQ  submission,  the  OA  Team  or  SLO  may  receive  additional 
requests  for  information  or  clarification  from  HQJOC  or  the  OPE  Team.  There  should  also 
be  an  opportunity  to  comment  on  drafts  of  the  final  CA  report  before  it  is  briefed  to 
CJOPS.  These  drafts  will  provide  an  opportunity  to  gain  consensus  on  the  weighting  of 
information  from  various  sources,  understand  the  impact  of  the  broader  context  on  the 
interpretation  of  information  provided  by  the  deployed  HQ  and  comment  on  the  validity 
of  analysis  feeding  in  to  the  final  report.  Timeframes  for  this  collaborative  feedback  loop 
are  often  compressed  by  HQJOC,  depending  on  HQJOC's  scheduling  of  the  CA  Board  or 
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the  timing  of  subsequent  briefings  to  the  Defence  Strategic  Command  Group  or  the 
Government's  National  Security  Committee. 

4.2.4  Implementation  Strategy 

In  preparing  for  a  CA  cycle,  or  when  conveying  lessons  on  a  previous  cycle,  consideration 
should  be  given  to  any  tangible  obstacles  to  execution  of  this  task.  This  includes  any  time 
constraints  or  concurrency  issues;  identification  of  critical  enablers  such  as  staffing, 
resources,  IT  systems  security,  or  ADF  support. 

4.2.4.1  Staff  and  skill  sets 

The  core  skill  set  of  deployed  analysts  for  the  execution  of  CA  as  identified  by  previous 
OA  Team  and  fly-away-team  members  are  liaison,  interviewing  techniques,  initiative, 
confidence,  the  ability  to  establish  contacts  and  a  willingness  to  communicate.  Any 
additional  skills  that  individual  analysts  may  possess  can  be  considered  a  bonus  (Hickman 
2011).  Previous  OA  Teams  have  raised  command  and  control  issues  in  relation  to 
appropriate  channels  for  reaching  into  theatre  to  gain  CA  information.  For  example,  it  may 
not  be  appropriate  for  HQJOC  to  reach  directly  into  the  Operation  Slipper  OA  team  for 
information  without  first  going  through  the  deployed  HQ  J8  officer.  This  issue  will  vary 
between  rotations,  depending  on  operational  tempo  and  the  preferences  of  various 
individuals  in  a  chain  of  command.  It  is  recommended  that  appropriate  processes  for 
information  requests  be  discussed  prior  to  the  commencement  of  CA  cycles  (e.g.  that  the  J8 
be  notified  of  subsequent  correspondence). 

4.2.5  Learning  Events 

In  the  interest  of  achieving  better  quality  assessments  and  in  making  the  CA  process 
easier,  the  OA  Team  is  advised  to  review,  capture  and  communicate  lessons  on  the  process 
after  the  completion  of  each  cycle. 

The  kinds  of  input  from  the  deployed  HQ  that  the  OA  Team  are  able  to  capture  at  this 
stage  are: 

•  Have  any  gaps  been  identified?  Is  the  metric  too  broad  or  too  narrow  so  that  it  fails 
to  provide  an  opportunity  to  report  on  the  contribution  of  an  area  to  achieving  its 
Effects? 

•  What  information  sources  was  given  primacy  in  responses,  and  why?  Are  there 
other  perspectives  that  could  be  considered? 

•  What  boundaries  or  constraints  are  in  place  within  the  response,  and  why?  Are 
these  transparent  for  analysts? 

•  Is  there  a  clear  understanding  of  the  delineation  between  MoPs  and  MoEs? 

•  Are  the  criteria  and  rating  scales  understood? 

•  Are  the  degrees  of  validity  of  source  data  made  clear? 
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•  If  there  is  insufficient  information  to  respond  to  a  given  MoE,  have  you  identified 
this  information  gap? 

•  Are  there  any  comparative  assessments  that  can  be  made  relative  to  other 
timeframes,  seasons,  locations,  data  sources? 

•  Does  the  CA  process  work?  How  can  it  be  improved? 


4.3  Evaluation  Plan 

The  differences  between  'monitoring'  and  'evaluation'  are  often  difficult  for  stakeholders 
to  appreciate.  Monitoring  is  only  useful  and  interesting  if  it  is  used  effectively  for 
evaluation.  Simply  charting  trends  or  listing  achievements  does  not  provide  insight  into 
whether  activities  are  combining  to  achieve  a  desired  Effect,  and  has  limited  utility  in 
determining  causal  factors  to  inform  decisions  on  future  courses  of  action.  Evaluation 
synthesises  monitoring  data  from  a  range  of  sources  and  verifies  it.  Where  inconsistencies 
appear  between  sources,  the  evaluation  process  acquires  further  information,  or  seeks  to 
better  understand  the  explanations  for  variation  in  interpretation  of  data.  In  many 
instances,  evaluation  of  a  particular  factor  will  not  result  in  any  change  in  the  conclusions 
reached  at  the  monitoring  stage,  and  will  simply  verify  them.  The  remainder  of  the  time 
though,  the  evaluation  stage  will  verify,  elaborate  on  and  contextualise  insights  drawn  out 
at  the  monitoring  stage.  In  this  way,  we  can  move  beyond  simply  documenting  what  has 
happened  to  date,  and  evaluation  insights  become  a  more  sound  basis  for  decision  making 
on  what  is  likely  and  desirable  for  the  campaign  plan  going  forward. 

4.3.1  Data  Collection  Strategy 

Throughout  the  quarter,  OPE  Team  will  be  responsible  for  collecting  data  relevant  to  each 
of  the  MoEs  and  to  the  operational  context  more  broadly.  This  may  be  assigned  as  the 
primary  responsibility  of  one  member  of  OPE  who  will  interrogate  daily  Defence 
reporting  and  store  any  relevant  data  in  the  files  created  for  each  of  the  twenty  effects  in 
the  shared  repository  on  Sharepoint  -  which  is  the  same  location  that  information  from  the 
deployed  OA  or  fly-away  team  should  be  stored.  ASSESSREPS,  INTREPS,  INTSUMS, 
deployed  HQ  SITREPS  and  periodical  coalition  reporting  are  envisaged  as  primary 
sources  of  CA-relevant  information.  If  done  properly,  they  should  summarise  everything 
relevant  to  CA  from  other  forms  of  reporting.  They  are  also  authorised  to  be  released  and 
therefore,  are  not  subjected  to  the  same  issues  as  sharing  raw  data  from  other  sources. 

At  the  same  time,  the  remaining  members  of  OPE  Team  working  on  the  CA  cycle,  will  be 
responsible  for  maintaining  awareness  of  open  source  reporting  and  significant 
developments  relevant  to  Australian  Defence's  role.  The  operational  context  is  everything 
other  than  direct  ADF  operational  activity.  In  relation  to  CA,  contextual  awareness  makes 
assessment  of  progress  against  the  Campaign  Plan  more  realistic.  It  allows  us  to  look 
beyond  our  own  immediate  perceptions  to  make  more  accurate  assessments  of  whether 
we  are  genuinely  succeeding  or  failing  in  the  eyes  of  the  local  population,  coalition 
partners,  adversaries  and  the  Australian  Government.  Also,  by  taking  account  of  the  full 
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spectrum  of  potential  causal  factors  for  progress  (or  not)  towards  achievement  of  Effects, 
we  are  better  positioned  to  analyse  how  our  activities  interact  with  the  local  context,  and 
can  then  make  assessments  of  what  activities  are  likely  to  be  most  effective  (Meharg  2009). 
This  means  not  just  asking  "What  works?",  but  "What  works  for  whom,  when,  where  and 
why?"  There  is  no  doubt  that  taking  account  of  contextual  information  adds  to  the 
complexity  and  analysis  burden  of  CA,  but  this  is  far  outweighed  by  increased  validity  of 
assessments  and  benefits  for  situational  awareness. 

Approximately  1  month  prior  to  the  end  of  the  assessment  period,  HQJOC  J5  (Plans) 
Branch  staff  will  seek  data  submissions  from  HQJOC  J2  (Intelligence),  Defence 
International  Policy  Division,  the  ADF  Special  Operations  Task  Group  and  other  areas  of 
Defence  on  specific  MoE  as  necessary.  It  will  always  be  necessary  for  DSTO  staff  to 
confirm  that  these  information  requests  have  been  sent  and  to  follow  up  on  inputs. 

A  fourth  and  final  component  of  the  data  collection  strategy  is  eliciting  input  from  other 
Government  agencies  (OGAs);  at  minimum  AusAID  and  the  Australian  Federal  Police. 
This  is  achieved  by  OPE  Team  members  and  HQJOC  J5  (Plans)  Branch  staff  through 
meetings  to  be  held  usually  in  Canberra  during  the  data  collection  phase.  Obviously  these 
meetings  will  need  to  be  arranged  well  in  advance,  but  they  will  be  driven  by  information 
gaps  and  questions  identified  by  the  analysis  team  during  review  of  other  data  and  initial 
drafting  of  the  report.  Data  from  these  meetings  should  be  captured  in  formal  written 
Records  of  Conversation.  These  meetings  are  also  an  opportunity  to  confirm  OGA 
attendance  at  any  scheduled  CA  Working  Group  (CAWG)  and  CA  Boards. 

4.3.2  Analysis  Strategy 

The  analysis  phase  of  the  assessment  cycle  will  require  the  nomination  of  an  OPE  Team 
lead.  The  team  leader  will  take  responsibility  for  re-confirming  agreement  on  forms  of 
support  and  timelines  with  the  HQJOC  J5  (Plans)  Branch  staff,  as  well  as  coordinating 
delivery  of  analysis  products  and  allocating  specific  analysis  tasks  to  supporting  team 
members,  ideally  matching  expertise  to  various  lines  of  operation. 

End  of  cycle  analysis  will  usually  commence  several  weeks  prior  to  receipt  of  the  deployed 
HQ  submission,  starting  with  OPE  Team  lead  consultation  with  HQJOC  J5  (Plans)  Branch 
staff  to  confirm  the  form,  scope  and  scale  of  assessment  required  for  this  quarter,  including 
any  areas  of  special  interest  to  be  featured  in  line  with  Strategic  Command  Group  or 
Cabinet  requirements.  At  this  stage,  timing  for  feedback  of  drafts  to  deployed  HQ  staff, 
any  CAWGs  and  the  CA  Board  will  be  agreed. 

4.3.2.1  Primary  Analysis 

In  the  weeks  prior  to  the  end  of  the  CA  cycle,  OPE  Team  members  will  be  allocated  a  Line 
of  Operation  to  start  working  through  the  data  stored  throughout  the  cycle  on  Sharepoint 
and  responses  to  MoEs  received  from  various  areas  of  Defence  as  they  are  received  from 
HQJOC  J5  (Plans)  Branch  staff.  Analysis  will  focus  on  drawing  out  the  following  aspects  as 
part  of  early  draft  sections  of  the  report: 

•  significant  events/ developments 
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•  trends  of  change  (positive  or  negative),  and  continuation  of  trends  identified  in 
previous  assessments. 

•  status  of  activities  forecast  in  previous  cycles  (both  ADF  and  other  stakeholders) 

•  constraints 

•  information  gaps 

•  inconsistencies 

•  additional  requests  for  information  (RFIs)  for  the  deployed  FIQ,  other  Defence  and 
intelligence  agencies,  and  questions  for  OGAs. 

The  draft  sections  of  the  report  developed  at  this  stage  start  to  construct  a  narrative  in  two 
potential  ways.  Firstly  it  may  centre  around  a  significant  event  or  milestone  and  use 
examples  and  reported  perspectives  to  articulate  possible  causal  factors.  Secondly,  it  may 
articulate  an  observed  trend  citing  evidence  of  that  trend  in  the  form  of  examples  of 
activities,  reported  observations  or  quantitative  analysis  of  criteria  associated  with  that 
trend.  In  both  cases  positive  or  negative  change  and  causal  links  must  be  explained  in 
terms  of  the  data  available  (i.e.  evidence-based),  ideally  referencing  reporting  from  more 
than  one  source. 

4.3. 2. 2  Secondary  analysis 

At  this  point,  the  team  has  a  substantial  working  draft  from  which  to  commence  analysis 
of  incoming  responses  to  the  MoE  set  from  theatre  and  elsewhere.  Each  Effect  can  now  be 
assessed  within  each  Line  of  Operation,  in  terms  of  progress  towards  Decisive  Conditions. 
Specifically  analysis  should  progress  beyond  simple  data  synthesis,  to  where  expert 
analysis  value  is  added  to  the  process  by  drawing  out: 

•  indicators  of  future  likelihoods 

•  emerging  risks  and  potential  mitigation  strategies 

•  assessments  against  planned  progress  and  implications  of  these 

•  planning  considerations 

•  recommendations  for  investigation  or  action. 

During  this  stage,  analysts  will  need  to  be  cautious  of  simplification  of  causal  chains  and 
failure  to  take  into  account  combined  complex  'causal  packages'.  Further,  DSTO  analysts 
must  maintain  a  clear  concept  of  the  boundaries  of  their  expertise  and  leave 
recommendations  on  military  planning  that  fall  out  of  analysis  for  J5  consideration. 

When  each  of  the  sections  of  the  draft  is  consolidated  by  the  team  leader  and  J5  staff  into  a 
single  document,  analysis  ceases  and  the  DSTO  team's  staff  effort  decreases  to  one  to  two 
staff  supporting  drafting  of  the  final  report.  This  is  likely  to  include  some  analysis 
response  to  issues  that  emerge  during  the  CAWG,  as  part  of  the  re-drafting  process 
described  under  Reporting  Strategy  below. 
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4.3.3  Reporting  Strategy 

As  detailed  as  an  initial  step  in  the  Analysis  Strategy  (4.3.2)  above,  it  is  crucial  that  there  is 
clarification  with  HQJOC  J5  (Plans)  Branch  staff  of  the  required  reporting  format,  prior  to 
analysis  and  drafting,  and  this  format  and  scale  should  be  maintained  unless  changes  are 
jointly  agreed.  This  includes  differentiating  between  a  "full  report"  and  a  smaller  "update" 
for  quarterly  CAs.  For  example,  there  may  be  reconsideration  of  the  scale  of  an  assessment 
if  it  falls  during  a  key  decision  point  period,  or  follows  significant  events  in  the  area  of 
operations. 

The  factors  that  should  be  considered  in  the  scoping  of  the  report  with  HQJOC  J5  (Plans) 
Branch  staff  include: 

•  Timeliness  -  what  is  achievable  in  the  time  available  to  provide  input  to  decision¬ 
makers  (eg.  what  are  the  imminent  decision  windows,  key  planning  events  and 
Government  meetings?).  In  the  time  allowed,  what  will  be  good  enough? 

•  Is  the  target  audience  for  the  report  (ie.  those  who  are  best  placed  to  take  action  or 
learn  from  it)  primed  to  receive  and  understand  it?  Will  key  players  be  available  to 
be  briefed  on  it,  and  in  turn  brief  their  seniors? 

•  Is  the  report  structured  to  convey  clear  messages,  with  key  areas  of  interest  easily 
accessible? 

•  Is  there  a  word/ page  limit  on  the  report?  The  balance  between  the  reasonable 
lengths  of  the  document  and  the  depth/ comprehensiveness  of  coverage  for  each 
MoE  should  be  considered. 

This  consensus  on  reporting  form  and  content  will  drive  the  analysis  and  drafting  stage 
detailed  in  Section  4.3.3,  and  should  result  in  the  production  of  a  consolidated  CAWG 
draft  assessment  that  is  circulated  to  staff,  including  the  deployed  HQ,  for  review  prior  to 
the  meeting.  At  some  stage  prior  to  the  meeting,  separate  briefings  may  be  held  with 
primary  stakeholder  representatives  (such  as  International  Policy  Division)  if  assessments 
are  likely  to  have  a  major  impact  on  their  programs.  Any  concerns,  queries  or  changes 
agreed  upon  by  the  CAWG  are  then  addressed  through  a  re-drafted  assessment  report 
which  is  prepared  for  the  J5  to  present  to  CJOPS  at  the  CA  Board. 

For  CA,  it  should  be  appreciated  that  the  value  of  the  assessment  is  not  limited  to  the 
production  of  formal  written  reports.  The  CA  process  itself,  including  the  presentation  of 
the  reports  at  CAWGs  and  CA  Boards,  provides  a  cue  for  discussion  and  frequently 
reveals  assumptions,  disconnects  and  underlying  risks  which  can  be  addressed  as  part  of 
the  informal,  but  still  important  outcomes  of  CA. 

4.3.4  Implementation  Strategy 
4.3.4.1  Costs 

Under  the  previous  OPSTSR  132,  and  now  under  the  on-going  Science  and  Technology 
Support  Request  (STSR),  DSTO  will  cover  the  cost  of  Australian-based  staff  for  this  work, 
and  will  assume  responsibility  for  most  of  the  second  order  costs  associated  with  this 
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program,  as  detailed  in  the  STSR  response  plan  (DSTO  2012).  HQJOC  agreed  to  fund 
domestic  travel  for  DSTO's  support  from  1  January  2012. 

4.3. 4.2  Staff  and  skill  sets 

It  was  previously  estimated  that  DSTO  would  be  able  to  support  approximately  three  CA 
fly-away-teams  per  year  to  any  of  the  three  primary  areas  of  ADF  operation  (Afghanistan, 
East  Timor  and  The  Solomon  Islands).  However,  DSTO's  ability  to  provide  a  high 
standard  of  support  to  these  requests  is  contingent  on  being  staffed  to  a  level  that  allows 
the  team  to  engage  in  enabling  research  on  the  science  of  monitoring  and  evaluation,  and 
the  growth  of  country-specific  expertise  for  continuous  data  gathering  on  each  area  of 
operation  in  between  CA  cycles  and  inferring  meaning  appropriate  to  environmental 
context. 

4.3. 4.3  Access  to  IT  Equipment  &  Networks 

The  quality  and  timeliness  of  support  is  contingent  on  access  to  IT  resources  and  facilities, 
such  as  network  access  to  both  DSTO  C-NET  and  DSN.  Current  reliance  on  WYSE 
terminals  for  remote  connection  to  classified  networks  is  unreliable  and  slow  given  the 
size  of  data  and  documents  being  stored,  the  requirement  to  switch  between  various 
documents,  and  requirement  for  the  use  of  data  visualisation  tools. 

4.3. 4.4  Client  engagement 

The  ability  of  DSTO  to  meet  requirements  is  also  contingent  on  investment  of  time  and 
staff  effort  from  HQJOC  to  provide  on-going  guidance  and  willingness  to  participate  in 
CA  scoping,  analysis,  drafting  and  review  activities.  This  level  of  engagement  will 
empower  OPE  Team  and  OA  Teams  to  allocate  staff  time  and  direct  their  efforts  to  best 
meet  the  evolving  needs  of  senior  command  and  planners. 

4.3.5  Learning  Events 

The  CA  process  has  two  scheduled  learning  events:  the  CAWG  and  CA  Board,  which  are 
intended  to  be  held  every  cycle.  The  CAWG  is  held  at  the  SOI  staff  level  within  HQJOC, 
but  also  includes  representatives  from  1st  Division,  International  Policy  Division,  and 
OGAs.  It  provides  and  opportunity  for  participants  to  review  a  draft  CA  report,  provide 
corrections  and  gain  an  understanding  of  potential  recommendations.  The  report  is  re¬ 
drafted  as  a  result  of  the  CAWG  before  being  submitted  for  the  deployed  HQ  review,  and 
then  re-circulated  at  the  J-staff  and  equivalent  level  for  consideration  before  being 
presented  by  CJOPS  at  the  CA  Board.  The  CA  Board  is  an  opportunity  for  questioning  the 
report's  findings,  consideration  of  the  planning  implications  of  those  findings  and  for 
CJOPS  to  agree  or  disagree  with  recommendations  and  direct  action. 

The  extent  of  influence  of  these  evaluations  is  not  limited  to  producing  quarterly  reports. 
The  commitment  of  an  organization  to  genuine  evaluation  is  reflected  in  the  extent  to 
which  they  include  it  in  their  decision-making,  and  in  turn  draw  upon  it  as  part  of 
planning  processes.  The  fact  that  CA  resides  within  the  HQJOC  J5  (Plans)  Branch  partially 
achieves  this,  and  has  led  to  CA  analyst  participation  in  reviews  of  campaign  plans,  and 
reviews  of  campaign  Effects  and  the  MoE  sets  that  underpin  them. 
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To  enable  continuous  improvement  of  the  CA  process  itself,  OPE  Team  are  responsible  for 
hosting  ex-OA  Team  member  lessons  workshops,  to  reflect  on  how  the  process  can  be 
improved,  and  derive  lessons  for  future  raise  train  and  sustainment  of  OA  capability. 


5.  Findings  from  Application  of  the  Process 

DSTO  experience  in  applying  this  evaluation  framework  to  the  provision  of  support  over 
successive  CA  cycles11  has  resulted  in  amendments  to  the  process,  and  a  number  of 
significant  insights  for  the  conduct  of  CA,  and  for  program-level  evaluations  in  general. 
These  are  briefly  discussed  here. 

5.1.1  Authority  to  conduct  evaluation 

CAs  conducted  on  all  three  operations  have  demonstrated  that  coordination  between 
deployed  and  Australian-based  analysts  can  become  problematic,  if  the  roles  and 
responsibilities  to  conduct  evaluation  are  not  understood.  There  is  a  demonstrated  need 
for  articulation  of  who  has  the  authority  to  execute  the  process  and  what  decisions  are 
informed  by  CA.  The  role  of  DSTO  is  to  support  CA,  not  to  own  it  or  task  deployed  HQ 
staff  to  execute  the  process.  The  necessity  for  CA  and  the  responsibility  to  execute  it  as  part 
of  core  operational  functions  must  be  directed  by  CJOPS  and  actioned  by  the  deployed  HQ 
Commander  (Defence  2011).  DSTO  can  then  be  empowered  to  assist  with  developing 
frameworks  and  processes,  as  well  as  setting  up  educational  programs  or  tools  in  order  to 
enable  the  ADF  to  execute  the  CA  process. 

Within  the  framework  of  the  CA  support  process  presented  here,  the  initial  phase  of 
establishing  the  Context  for  Evaluation,  is  the  logical  place  for  articulation  of  the  authority 
to  conduct  an  evaluation.  However,  this  has  not  yet  been  executed  in  CA  cycles,  due  to  the 
constraints  discussed  in  Section  2.3,  such  as  execution  of  CA  on  already  mature  plans,  and 
operational  time  constraints.  This  guidance  is  partially  contained  in  the  current  Standing 
Instruction  (OPS)  on  Campaign  Assessment  (Defence  2011),  but  this  does  not  detail  the 
type  of  information  required  and  how  it  is  to  be  collected.  It  has  been  previously 
suggested  during  CA  process  review  workshops  that  some  form  of  formalised  initiation  of 
each  cycle  from  HQJOC  to  Commander  of  the  deployed  HQ  is  required.  This  could 
eliminate  potential  misunderstandings  over  submission  dates  and  enable  deployed  HQ 
staff  to  be  directed  to  devote  effort  to  the  task,  as  well  as  to  authorise  the  DSTO  OA  Team's 
work  with  deployed  HQ  staff  to  achieve  quality  deployed  HQ  input  into  the  process  (e.g. 
requests  for  deployed  HQ  intelligence  product). 


11  Evidence  and  discussion  of  all  CA  cycles  to  date  are  contained  in  classified  draft  DSTO  reports  on  support 
to  operations  in  Afghanistan,  East  Timor  and  The  Solomon  Islands,  which  can  not  be  referenced  here. 
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5.1.2  Coordination  between  deployed  and  AUS-based  analysts 

In  order  to  provide  a  valid  assessment  of  the  current  operating  environment,  CA  is  reliant 
on  significant  input  from  deployed  HQs.  However,  to  situate  that  assessment  in  the 
context  of  the  local  culture,  the  activities  of  other  stakeholders,  and  to  verify  ADF  insights 
against  other  reporting  perspectives,  CA  is  also  dependent  on  analysts  with  time, 
theoretical  expertise  and  resources  to  synthesise  the  Deployed  HQ  inputs  into  strategic- 
level  assessments.  This  presents  challenges  for  communication  of  intent,  findings  and 
explanation  of  decisions  made  as  part  of  the  translation  of  data  into  a  HQJOC  report.  The 
potential  for  divergence  of  approach,  misinterpretation  and  disenfranchisement  needs  to 
be  a  constant  consideration  from  start  to  finish  of  the  CA  process.  Shared  information 
management  and  constant  communication  are  both  potential  mitigations  for  this  risk. 

5.1.3  Information  management 

CA  requires  the  constant  monitoring  and  synthesis  of  massive  amounts  of  data  in  diverse 
formats.  Information  management  systems,  data  synthesis  tools  and  pre-deployment 
training  of  DSTO  OA  teams  need  to  be  included  as  components  of  the  process  to  ensure 
that  the  deployed  HQ  is  able  to  meet  data  requirements  of  each  CA  cycle.  Without  this 
consideration  the  quality  of  analysis  and  the  practicality  of  scientific  design  will  quickly 
become  compromised12.  For  this  reason,  it  has  been  consistently  recommended  that  OA 
Teams  and  deployed  HQ  staff  develop  an  awareness  of  the  MOEs  that  are  relevant  to  their 
area.  This  familiarity  with  MoEs  will  enable  them  to  continuously  record  any  data  that 
may  be  relevant  to  CA  into  a  some  form  of  repository,  so  that  it  can  be  routinely  added  to 
as  information  comes  to  light.  This  repository  can  then  be  easily  collated  when  the  CA 
submission  cycle  comes  around,  instead  of  staff  experiencing  a  spike  in  work  load  as  they 
try  to  collect  and  review  3-6  months  worth  of  data  at  CA  submission  time. 

This  is  sometimes  exacerbated  by  a  tendency  to  invent  new  databases  or  information 
management  system  file  structures  each  time  an  information  management  issue  is 
encountered.  While  there  are  opportunities  for  continuous  improvement  in  information 
management,  where  we  are  attempting  to  establish  sustainable  systems  that  can  survive 
OA  Team  and  deployed  HQ  rotations,  simply  making  do  with  inherited  systems  can  save 
time  and  effort  and  allow  the  establishment  of  consistent  baselines  for  analysis. 

Information  management  systems  need  to  be: 

•  easily  accessible 

•  compatible  with  deployed  IT  systems 

•  suitable  for  monitoring  of  variables  on  a  continuous  basis 

•  searchable  to  allow  for  extraction  of  data  across  files,  for  ease  of  aggregation. 


12  As  noted  by  Sue  C.  Funnell  &  Patricia  J.  Rogers  (2011)  Purposeful  Program  Theory:  Effective  use  of 
theories  of  change  and  logic  models,  Jossey-Bass,  San  Francisco,  USA. 
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5.1.4  Staff  and  skill  sets 

For  both  monitoring  and  evaluation  plans,  the  quality  of  CA  is  contingent  upon  the 
availability  of  adequate  numbers  of  suitably  qualified  analytical  staff.  The  provision  of 
quality  DSTO  support  to  CA  has  been  significantly  impaired  in  the  past  by  the  lack  of 
availability  of  staff,  combined  with  the  time  constraints  discussed  in  section  4.1. 

Implementation  of  the  monitoring  component  of  the  evaluation  framework  by  successive 
OA  Teams,  has  confirmed  that  pre-deployment  education  on  the  importance  and  aims  of 
CA  improves  the  deployed  HQ  contribution.  The  most  effective  preparation  to  date  has 
included  briefings  at  pre-deployment  courses,  combined  with  several  weeks  embedded 
with  DSTO  evaluation  analysts.  This  type  of  preparation  has  built  familiarity  with  the  CA 
tasks  required  of  OAs  when  deployed,  but  has  also  established  valuable  relationships 
between  deployed  and  Australian-based  evaluation  analysts,  which  opens  channels  for 
communication  of  data,  resolution  of  obstacles  and  collaborative  analysis.  Successive  CA 
cycles  have  also  confirmed  that  the  ideal  OA  skill  for  the  execution  of  CA  includes  liaison, 
interviewing  techniques,  initiative,  confidence,  the  ability  to  establish  contacts  and  a 
willingness  to  communicate. 

It  has  become  apparent  that  the  execution  of  the  evaluation  component  of  CA  also  requires 
staff  with  a  particular  skill  set.  Firstly,  the  ability  to  think  strategically  by  processing 
information  in  the  context  of  it  implications  for  all  other  variables  in  the  campaign  plan. 
Secondly,  experience  in  making  sense  of  high  degrees  of  uncertainty,  and  in  deriving 
evidence-based  conclusions  from  large  amounts  of  diverse  data.  This  requires  the  ability  to 
maintain  focus  on  the  agreed  analytical  approach,  resilience  to  overwhelming  quantities  of 
data  at  various  stages  of  the  process,  and  experience  in  identifying  and  prioritising  trends. 
Thirdly,  a  sound  understanding  of  the  history  and  dynamics  of  ADF  campaign  planning 
and  CA,  is  essential  for  using  discretion  in  accommodating  changing  client  requirements, 
and  for  producing  practical  analysis.  Finally,  evaluation  analysts  will  ideally  have  existing 
knowledge,  or  the  ability  to  quickly  acquire  knowledge,  on  the  particular  operational 
environment,  such  as  country-specific  expertise  and  understanding  of  the  history  of  the 
operation. 


6.  Conclusions  and  Future  Work 


This  report  documents  guidance  for  DSTO  analysts  on  the  entire  process  for  providing 
support  to  HQJOC's  CA  for  operations  in  Afghanistan,  East  Timor  and  The  Solomon 
Islands.  It  application  in  each  of  these  contexts  has  been  successful  in  meeting  client 
requirements,  but  has  reinforced  the  necessity  for  a  single,  consistent,  rigorous  approach  to 
CA  by  DSTO  analysts  in  Australia  and  on  deployment.  The  components  of  this  process  are 
aimed  at  ensuring  this  consistency,  and  creating  a  cohesive  sequence  of  steps  from  start  to 
finish.  It  defines  the  boundaries  of  DSTO  support,  and  details  expected  DSTO  roles  and 
the  analytical  steps  that  are  required. 
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By  breaking  down  the  steps  required  for  support  to  CA,  it  has  become  evident  that  CA 
support  involves  the  collection,  management  and  analysis  of  very  large  amounts  of  data. 
Therefore,  the  CA  process  can  not  be  improved  by  simply  increasing  the  number  of 
analysts,  and  a  solution  has  to  be  found  for  more  efficient  processing  of  an  expanded  data 
set,  either  through  analysis  techniques  or  information  management  tools.  Notwithstanding 
this,  as  noted  in  Sections  4.3.4.2  and  5.1.4,  a  critical  mass  of  DSTO  staff  in  terms  of  both 
quantity  and  skill  set  is  imperative  for  providing  support. 

As  each  instance  of  support  to  CA  has  been  for  existing  operations,  it  has  not  been  possible 
to  optimise  the  format  of  campaign  planning  for  evaluation  purposes.  It  has  also  not  been 
possible  during  active  operations  to  consult  extensively  with  HQJOC  staff  to  explore  the 
Effects  defined  in  the  campaign  plan,  and  to  derive  a  MoE  set  and  any  additional 
evaluation  questions  for  those  Effects.  As  a  result  the  CA  process  is  often  'making  do'  with 
retro-fitting  an  evaluation  framework  to  a  given  campaign  plan  structure,  and  monitoring 
progress  against  Effects  and  Objectives  for  which  intent  can  only  be  inferred.  The  CA 
process  is  therefore  flexible  enough  to  fit  around  existing  operations,  and  those  who  apply 
it  must  be  reminded  that  it  is  not  intended  to  represent  best  practice  in  scientific 
methodology.  Rather,  it  is  a  practical  solution  based  in  scientific  theory,  for  immediate 
application,  but  also  continuous  improvement. 


6.1  Future  Work 

In  order  to  address  a  number  of  limitations  noted  above,  and  to  continue  to  enhance  the 
CA  framework,  a  number  of  options  for  future  work  are  proposed. 

As  discussed  in  Section  3.5,  there  is  on-going  research  being  conducted  on  the 
development  and  implementation  of  the  Logic,  Assumptions  and  Risks  Framework 
(LARF).  Formalisation  of  the  LARF  as  a  component  of  the  CA  process  could  significantly 
enhance  the  monitoring  of  planning  assumptions,  which  will  provide  a  transparent  basis 
for  selecting  indicators.  The  LARF  also  has  the  potential  to  address  the  problem  of 
applying  evaluation  frameworks  to  existing  operations  by  providing  a  mechanism  for 
drawing  out  more  detailed  articulation  of  the  logic  that  underpins  campaign  plans.  This 
would  serve  as  a  reference  point  for  assessments  of  whether  operations  are  progressing 
along  intended  paths. 

It  is  apparent  from  the  findings  of  the  application  of  the  CA  process  that  DSTO's  support 
can  be  further  enabled  with  some  form  of  formalised  initiation  of  each  cycle  from  HQJOC 
to  the  Commander  of  the  deployed  HQ.  Further  consultation  with  HQJOC  and  deployed 
HQs  are  required  to  establish  the  optimal  way  to  formalise  the  authority  and  roles  of  CA. 
This  would  set  expectations  of  deliverables  and  to  authorise  the  DSTO  OA  Team's  work 
with  deployed  HQ  staff  to  achieve  quality  deployed  HQ  input  into  the  process  (e.g. 
requests  for  deployed  HQ  intelligence  product). 

The  process  through  which  DSTO  provides  support  to  HQJOC  CA  should  be  subject  to 
continuous  improvement,  with  a  number  of  specific  improvements  highlighted  here. 
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However,  the  process  has  been  of  significant  benefit  to  Defence  operations,  described  by 
Deputy  CJOPS  (Johnston  2011)  as  follows: 

An  important  new  capability  has  now  been  established  to  provide  the  analytical 
support  to  effects-based  Campaign  Assessments.  Campaign  Assessments  for 
Afghanistan,  East  Timor  and  the  Solomon  Islands  are  now  able  to  inform  Chief  of 
Joint  Operations,  Chief  of  Defence  Force  and  the  Strategic  Command  Group  of 
progress  toward  operational  and  strategic  objectives  and  provide  a  robust 
mechanism  for  conditions-based  decision  making. 

This  report  therefore  provides  guidance  on  the  purpose,  scope  and  steps  of  a  process 
through  which  DSTO  can  provide  significant  support  to  ADF  operations. 
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Appendix  A: 


Example  of  campaign  assessment  analysis  sequencing 


Campaign  Assessment 

Week  1 

\  Week  2 

Week  3 

\  Week  4 

^  Week  5 

'S  Week  6 

^  Week  7 

Week  8 

Week  9 

^Week  1 0 

^A/Veek  1 1 

^A/Veek  12 

^Week  1 3 

^fc/eek  14 

^fc/eek  15 

> 

v  Week 
?  16 

JOC 

Issue  a  Warning  Order  or 
Task  Order  to  JTF  to  initiate 
process. 

Refine  MoE  set  &  template  & 
send  to  OA  Team,  relevant 
Defence  areas  &  OGAs 

Commence 
draft  CAWG 
paper 

Incorporate  responses  from  OGAs  and  Defence  areas  as 

received. 

Arrange 
CAWG  with 
WoG  invitees. 

Distribute  draft 
paper  to 
CAWG 
members 

Attend  CAWG, 
&  capture 
conclusions  & 
additional  RFIs 

Draft  CAB 
document 

Distribute  CAB 
draft  document 
for  comment 

Briefs  & 
presentat.s 
for  J5, 
CJOPS  & 
CAB 

Brief  J5, 
CJOPs  and 
attend  CAB 

Incorporate 

CAB 

comments 

in 

preparation 
for  brief  to 
SCG 

DSTO 

Refine  MoE  set  &  template  & 
send  to  OA  Team,  relevant 
Defence  areas  &  OGAs 

Respond  to 
feedback  from 
OA  Team  & 
change  MoE 
template. 

Support  drafting  CAWG  paper,  incorporating  all  INT,  JTF  & 
open  source  reporting  from  this  cycle  &  respond  to  reachback 
OA  team  requests. 

Verify  and  align  assessments 
&  incorporate  JTF  responses 
into  CAWG  paper. 

Attend  CAWG 
&  incorporate 
conclusions  in 
draft  paper 

Draft  CAB 
document 

Incorporate 
comments 
received  on 
draft  CAB 
document 

Brief  J5, 
CJOPs  and 
attend  CAB 

Back  brief 
to  OA  Team 
and 

recommenc 

e  research 
for  next 
cycle 

Deployed  OA 

For  Op  Slipper, 
there  are  2  OA 
Teams:  AMAB 
and  TK. 

AMAB  is 
responsible  for  all 
aspects  of  the  CA 
and  requests 
support  from  TK 
as  required. 

Examples  of 
where  this 
support  may  be 
required  are 
denoted  by:  (TK?) 

CA 

awareness 
workshops  & 
socialise  MoE 
set  &  provide 
feedback  to 
JOC  JE  Team 

Assist  MoP 
data  gathering 
&  analysis  & 
work  with  J2  to 
include  MoEs 
in  PIRs 

Distribute  MoE  template, 
assist  MoE  data  gathering 
(TK?)  and  commence  drafting 
of  JTF  report 

Collect 
responses 
(TK?)  & 
produce  draft 
report 

Workshop 
draft  with  all  J 
staff  (TK?) 

CJTF  approval 
of  draft  &  send 
to  JOC 

Respond  to 
JOC  requests 
for  clarification 
of  data.  (TK?) 

Coordinate 
response  to 
CAWG  RFIs 

Provide 
comment  on 
draft  CAB 
document 

Brief  CJTF 
prior  to  CAB 

Back  brief 
to  JTF  & 
review 
process  for 
next  cycle 

Continuous  collection  of  data  against  MoEs  as  part  of  daily  reporting  and  information  management 

Ll_ 

h- 

— 5 

On  receipt  of 
WO  or  TO, 
advise  OA 
Team  and 
direct 

execution  of 
CA. 

Attend  CA 

awareness 
workshops  & 
provide 
feedback  to 
OA  Team  on 
applicability  of 
MoEs 

J2  to  consider 
mapping  of 
MoEs  to  PIRs 

Map  MoP  and  other  activity 
data  to  MoEs 

Aggregate 
MoE  related 
data  and 
supply  to  OA 
Team 

Review  draft 
JTF  report 

Command 
staff  briefed  on 
JTF  report  and 
CJTF  sign  off 

Assist  OA 
Team  in 
response  to 
JOC  requests 
for  clarification 

Assist  OA 
Team  in 
response  to 
CAWG  RFIs 

Contribute  to 
OA  Team 
coordination  of 
comments  on 
draft  CAB 
document 

CJTF  briefed 
prior  to  CAB 

Provide 

feedback 

on 

process  to 
OA  Team 
for  next 
cycle 

OGAs 

Receive  and  consider  MoEs  from 
JOC 

Provide  responses 
on  relevant  MoEs 
to  JOC 

Meet  with  JOC  and  DSTO  staff  as 
necessary  to  discuss  progress  and 
available  data 

Attend  CAWG 

Receive  CAB  report,  inform  JOC  of 
any  queries  and  brief  CAB  attendees 

Attend  CAB 
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